Integration of Flux as a wrapper engine #2708

pablogoitia · 2025-11-13T12:12:06Z

Partially closes #2697

Check List

I have read CONTRIBUTING.md.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to pyproject.toml.
Tests are included (or explain why tests are not needed).
Changelog entry included in CHANGELOG.md if this is a change that can affect users.
Documentation updated.
If this is a bug fix, PR should include a link to the issue (e.g. Closes #1234).

…endency error The error occurred while executing "autosubmit run"

By now it is only functional with Slurm

It is pending to develop some remaining kinds of wrapper and incorporating error handling, retries, stat files...

…common script

…age builder Also added some clarifying comments for myself (marked as TODOs)

…latform

codecov-commenter · 2025-11-14T15:55:30Z

Codecov Report

❌ Patch coverage is 47.20000% with 66 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.48%. Comparing base (57d5cf1) to head (8469766).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
...tosubmit/platforms/headers/fluxoverslurm_header.py	27.08%	35 Missing ⚠️
autosubmit/platforms/wrappers/wrapper_factory.py	30.00%	10 Missing and 4 partials ⚠️
autosubmit/platforms/wrappers/wrapper_builder.py	60.00%	12 Missing ⚠️
autosubmit/job/job.py	66.66%	1 Missing and 1 partial ⚠️
autosubmit/scripts/autosubmit.py	0.00%	1 Missing and 1 partial ⚠️
autosubmit/platforms/fluxoverslurm.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2708      +/-   ##
==========================================
- Coverage   67.70%   67.48%   -0.23%     
==========================================
  Files          86       88       +2     
  Lines       19786    19894     +108     
  Branches     3840     3860      +20     
==========================================
+ Hits        13396    13425      +29     
- Misses       5458     5533      +75     
- Partials      932      936       +4

Flag	Coverage Δ
fast-tests	`67.48% <47.20%> (-0.22%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Some parameters are now directly provided to "flux batch"

manuel-g-castro

Overall I am really happy with the implementation.

I find it very elegantly done, given the high degree of difficulty of the code base.

I made some minor comments here and there, but my major two comments regard the creation of the STAT files and a file to check if the wrapper failed in the horizontal-vertical case.

The first point could turn to be unnecessary if @kinow 's PR stops the creation of this file.

The second point is a suggestion just to avoid yet-another-file in the parallel file system.

Whenever I have time, I plan to install this branch and try the wrappers.

Thanks!

autosubmit/platforms/wrappers/wrapper_builder.py

manuel-g-castro · 2025-11-20T14:37:18Z

autosubmit/platforms/wrappers/wrapper_builder.py

+        {1}
+
+        # Instantiate Flux within the allocated resources and run the jobs
+        srun --cpu-bind=none flux start --verbose=2 /usr/bin/bash flux_runner.sh


It would be nice to add a #TODO to parametrize the verbosity of Flux.

autosubmit/platforms/wrappers/wrapper_builder.py

pyproject.toml

…le flags

pablogoitia · 2025-11-21T13:51:39Z

Hi, @manuel-g-castro!

In commit 0823ab3 I introduce a new key to the Autosubmit wrapper configuration called CUSTOM_ENV_SETUP that enables to introduce the commands that the wrapper must execute to setup the remote Flux environment before calling flux start.

This new parameter is optional and can be assigned to a wrapper global or locally. That is, you can set the CUSTOM_ENV_SETUP value to the WRAPPER key and it will be applied to all the wrappers, or you can set it to a specific wrapper and will only affect it.

Here is an example:

WRAPPERS:
  WRAPPER_SIM:
    POLICY: "flexible"
    MIN_WRAPPED: 1
    MAX_WRAPPED: 6
    TYPE: "vertical"
    JOBS_IN_WRAPPER: "SIM"
    METHOD: FLUX
  CUSTOM_ENV_SETUP: |
    module load miniconda
    source /apps/GPP/MINICONDA/24.1.2/etc/profile.d/conda.sh
    conda activate flux
    conda info

I'm looking forward to know your experience loading the Spack package to see if Flux wrappers will work in any scenario.

I have also uploaded my workflow for testing vertical wrappers with the Flux method to WorkflowHub (it is private and only users under the BSC-CES organization can access it), in case you want to make a quick test.

…emented yet

pablogoitia added 8 commits November 13, 2025 12:56

Remove duplicated platform attribute

483f9a3

Temporaly downgraded the yaml version required to solve a runtime dep…

e73e1f2

…endency error The error occurred while executing "autosubmit run"

Temporaly added an entry point for debugging in VSCode

46add63

Integrated the Flux builder to the factories

f197f9c

By now it is only functional with Slurm

Created a wrapper builder for Flux (as wrapper engine)

84f79ce

It is pending to develop some remaining kinds of wrapper and incorporating error handling, retries, stat files...

Modified the wrapper header to assign "bash" as the language for the …

fe3298c

…common script

Just added some clarifying comments for myself (marked as TODOs)

223e00a

Now the wrapping method is passed as a parameter to the vertical pack…

5ec796e

…age builder Also added some clarifying comments for myself (marked as TODOs)

pablogoitia self-assigned this Nov 13, 2025

pablogoitia added this to Autosubmit project Nov 13, 2025

pablogoitia added enhancement New feature or request working on Someone is working on it labels Nov 13, 2025

github-project-automation bot moved this to Todo in Autosubmit project Nov 13, 2025

pablogoitia moved this from Todo to In Progress in Autosubmit project Nov 13, 2025

pablogoitia added 7 commits November 13, 2025 15:56

Basic horizontal wrapper handling with Flux

7d23980

Added some error handling to the builder (not the wrapper script)

0993b9c

Minor changes in comments

14e8e1b

Fixed exception message and horizontal wrapper generation for Flux

1c8e0ab

Temporaly associate jobs in wrapper to the FluxOverSlurm fictitious p…

c62199f

…latform

Create a fictitious platform for Flux running over Slurm

9911398

Now jobs inside a wrapper contains more wrapper information

03c1cb0

pablogoitia force-pushed the flux-wrapper-engine branch from 374fbdd to 03c1cb0 Compare November 14, 2025 16:02

pablogoitia added 7 commits November 14, 2025 19:12

Header format improvements

b4f8595

Forced nslots in the flux batch command

146026d

Cleaned Flux script header

3ccc0b5

Some parameters are now directly provided to "flux batch"

Finished the vertical and horizontal wrappers. Improvements in code.

80e9324

Comment fix

3d77764

Solved "Remove extraneous f prefix"

8469766

Now processors numbers are get per section to support V-H and H-V

a9cbd2c

pablogoitia added 5 commits November 18, 2025 16:19

Fix environment loader

b77c887

Support for vertical-horizontal wrappers

246b403

Implemented H-V wrappers and fixed a bug in V-H

787ab6f

Fixes in Flux headers

650eb6d

Added some TODOs

ddf2de0

manuel-g-castro reviewed Nov 20, 2025

View reviewed changes

pablogoitia added 6 commits November 21, 2025 11:14

Now jobs are required to be waitable through their headers

c337c00

Remove the non-existent tasks-per-node directive

65d50d5

Refactor the FluxWrapperBuilder to remove unused processor and waitab…

ee6240a

…le flags

Reorganize TODOs and remove unused commands

5f8a9db

TODO removal or update

5379ba2

Remove log path flags because they are overwritten in 'flux batch'

84082ac

pablogoitia force-pushed the flux-wrapper-engine branch from 52ad1d7 to 84082ac Compare November 21, 2025 11:26

Environment setup is now available as a config parameter

0823ab3

Now Autosubmit exits in case of receiving hetjobs, which are not impl…

b85501e

…emented yet

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Integration of Flux as a wrapper engine #2708

Integration of Flux as a wrapper engine #2708

pablogoitia commented Nov 13, 2025

Uh oh!

codecov-commenter commented Nov 14, 2025 •

edited

Loading

Uh oh!

manuel-g-castro left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

manuel-g-castro Nov 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pablogoitia commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Integration of Flux as a wrapper engine #2708

Are you sure you want to change the base?

Integration of Flux as a wrapper engine #2708

Conversation

pablogoitia commented Nov 13, 2025

Uh oh!

codecov-commenter commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

manuel-g-castro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

manuel-g-castro Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pablogoitia commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Nov 14, 2025 •

edited

Loading